Object detection device

ABSTRACT

The purpose of the present invention is to provide an object detection device conducive to carrying out control in an appropriate manner according to the surrounding environment, with consideration to the accuracy of locations of detected objects. The device is characterized by being provided with: a parallax information generation unit for generating parallax information from a plurality of parallax images acquired from a plurality of imaging units; an object detection unit for detecting objects contained in the parallax images; a location information generation unit for generating location information about the objects, on the basis of the parallax information; and a location accuracy information generation unit for generating location accuracy information pertaining to the accuracy of the location information, on the basis of the condition of generation of the parallax information.

TECHNICAL FIELD

The present invention relates to an object detection device, and anon-vehicle surrounding environment detection device that detects 3Dobjects present around a vehicle using a stereo camera, determineslikelihood of collision between the vehicle and the 3D objects on thebasis of the behavior of the detected 3D objects and the vehicle, andoutputs an alert or performs vehicle control.

BACKGROUND ART

The commercialization of an application for recognizing a vehiclesurrounding environment by a camera mounted on a vehicle is on theincrease. Among others, application to preventive safety technology forpreventing accidents from occurring or to vehicle control technologyaiming at autonomous traveling, by using the recognized object, has beenexpected. Recognition technology used for vehicle control naturallyneeds high reliability.

There has been PTL 1 as a technology in which, after a 3D object isdetected, the reliability for the detected object is achieved. Thedevice is an object detection device which detects a 3D object by usinga distance image generated by a stereo camera, and evaluates thetime-series stability or the contour shape of the detected objectaccording to evaluation measures to calculate reliability.

CITATION LIST Patent Literature

PTL 1: JP 2008-45974 A

SUMMARY OF INVENTION Technical Problem

As described above, in the conventional technology, reliability indetermining what an object is has been examined.

Meanwhile, reliability is also significant in addition to thedetermination of what an object is. For example, reliability or accuracypertaining to the position of the detected object is also significant.The conventional technology described above does not handle accuracypertaining to location information.

In view of this, the present invention aims to provide an objectdetection device conducive to carrying out control in an appropriatemanner according to the surrounding environment, with consideration tothe accuracy of locations of detected objects.

Solution to Problem

The present invention is characterized by being provided with: aparallax information generation unit for generating parallax informationfrom a plurality of parallax images acquired from a plurality of imagingunits; an object detection unit for detecting objects contained in theparallax images; a location information generation unit for generatinglocation information about the objects, on the basis of the parallaxinformation; and a location accuracy information generation unit forgenerating location accuracy information pertaining to the accuracy ofthe location information, on the basis of the condition of generation ofthe parallax information.

Alternatively, the present invention is characterized by being providedwith: a parallax information generation unit for generating parallaxinformation from a plurality of parallax images acquired from aplurality of imaging units; and an object detection unit that identifiesand detects a plurality of objects overlapping in an optical axisdirection of the imaging units, on the basis of a histogram for anamount of effective parallaxes projected on one axis, the effectiveparallaxes being parallaxes pertaining to objects to be imaged which areincluded in the parallax images and to which stereo matching has beenperformed.

Advantageous Effects of Invention

The present invention is conducive to carrying out control in anappropriate manner according to the surrounding environment, withconsideration to the accuracy of locations of objects.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an object detection device according to anembodiment of the present invention.

FIG. 2 is a block diagram of a pedestrian detection unit.

FIG. 3 is a block diagram of a location information generation unit.

FIG. 4 is a block diagram of an accuracy information generation unit.

FIG. 5 is a block diagram of a display/alert/control unit.

FIG. 6 is a diagram illustrating a method for determining likelihood ofcollision against a pedestrian.

FIG. 7 is a diagram illustrating a method for calculating currentpedestrian location accuracy from location accuracy of an instantaneousvalue.

FIG. 8 is a diagram illustrating a method for calculating pedestrianlocation accuracy after TTC second.

FIG. 9 is a diagram illustrating a control level table.

FIG. 10 is a diagram illustrating a control table based on predictedlocation accuracy and an offset width.

FIG. 11 is a diagram illustrating an outline of a process in a densitycalculation unit.

FIG. 12 is a diagram illustrating an outline of a process in aperipheral separation degree calculation unit.

FIGS. 13(A) to 13(C) are explanatory views of a scene in which an erroris likely to be caused on a pedestrian location.

FIG. 14 is an explanatory view for increasing location accuracy byutilizing a projection histogram.

FIGS. 15(a) to 15(c) are diagrams illustrating a method for analyzingaccretion.

FIG. 16 is a diagram illustrating a method for analyzing a light source.

FIG. 17 is a diagram illustrating a method for predicting the movementof a pedestrian.

FIG. 18 is a flowchart of a process for preventing collision against apedestrian.

DESCRIPTION OF EMBODIMENT

An embodiment of the present invention will be described below withreference to the drawings.

FIG. 1 illustrates the configuration of a pedestrian detection deviceusing a stereo camera. The pedestrian detection device includes a stereocamera imaging unit 100, a parallax image generation unit 200, apedestrian detection unit 300, a location information generation unit500, a location accuracy information generation unit 400, and adisplay/alert/control unit 600.

The stereo camera imaging unit 100 is provided with a stereo camera thatcaptures an image of an environment ahead of a vehicle. The parallaximage generation unit 200 generates parallax images by matching for eachof small areas in left and right images captured by the stereo camera.The pedestrian detection unit 300 extracts 3D objects by using theparallax images, and tracks the extracted 3D object candidates in atime-series manner. When 3D object candidates are stably extracted in atime-series manner, the pedestrian detection unit 300 determines whetheror not the parallax shape and the contour shape based on an edgeextracted from a current image seem to be a pedestrian. The locationinformation generation unit 500 generates the three-dimensional locationof the detected pedestrian. The location accuracy information generationunit 400 generates location accuracy information, which indicates theaccuracy of the location information, based on the result of theparallax images. The display/alert/control unit 600 executes display,alert, and control pertaining to preventive safety of the vehicle byutilizing the detection result, the location information, and thelocation accuracy information of the pedestrian.

FIG. 2 illustrates the detailed configuration of the pedestriandetection unit 300. A 3D object extraction unit 310 extracts a block ofparallaxes each being similar from the parallax images within arectangular frame, thereby extracting a contour of a 3D object whichseems to be one mass. Thus, an object from which parallaxes arecontinuously extracted in a stable manner can be extracted as a mass ofa 3D object. In addition, it is determined whether or not an object,which is at a short distance from the extracted 3D object candidate andhas similar parallax on the image, is to be combined. According to thisprocess, 3D object candidates which have been extracted as objectsdivided from an object, having many ineffective parallaxes because offailure in matching due to insufficient texture, are combined togetheras one mass. On the contrary, it is again searched whether or not apedestrian is included in the extracted 3D object candidates, and 3Dobject candidates including pedestrians, vehicles, motorbikes, etc. areextracted by the 3D object extraction unit 310.

Next, the 3D object extracted by the 3D object extraction unit 310 istracked by a tracking unit 320 in a time-series manner. The 3D object istracked in such a manner that the location of the 3D object, the size ofthe 3D object, and the predicted parallax value of the 3D object on animage of the current frame predicted from the previous frame inconsideration of the behavior of the vehicle are compared to similarlocations, sizes, and parallax values within a certain threshold, andthe similarity is compared between the location on the image of theprevious frame and the location on the image of the current frame.

If a 3D object is once recognized, high-precise tracking usingprediction information indicating that the target object is a vehiclemoving at certain kilometers per hour, for example, is enabled. However,at the beginning of the initial tracking, the movement of the targetobject is unclear, so that rough tracking is executed for a 3D object todetermine whether or not the 3D object is such an object which is stablyextracted in a time-series manner.

Next, a recognition unit 330 performs pedestrian determination for anobject with a pedestrian size in the 3D objects described above, therebydetermining whether or not the object is a pedestrian. At first, therecognition unit 330 briefly determines whether or not there is anobject, in 3D object candidates, having a pedestrian-like size, on theimage, according to a horizontal to vertical ratio and the depthdistance obtained through conversion from the parallax into thedistance. The recognition unit 330 does not perform the recognition fora 3D object with a size clearly different from the size as a pedestriancandidate, in consideration of the processing load.

Next, the recognition unit 330 determines whether or not the depth shapeof the 3D object with a pedestrian-like size in the parallax image seemsto be a pedestrian. For this process, a pattern matching method or atemplate matching method is used. In addition, it is roughly determinedwhether or not the pedestrian shape determined by using contourinformation in the original image seems to be a pedestrian, and a 3Dobject clearly different from a pedestrian is determined not to be acandidate for the recognition. This is because, since a recognizer forrecognizing a pedestrian has a high processing load, the narrowing ofcandidates to some extent is previously carried out to reduce theprocessing load as a whole for pedestrian recognition. The pedestriandetermination using the pedestrian recognizer is executed only for thepedestrian candidates which have been narrowed down by the pedestriandetermination performed in advance. The recognizer performs thepedestrian recognition on the basis of the distribution condition ofedge angles in a frame obtained by dividing the frame of the extracted3D object candidate in a lattice, and thus, it is finally determinedwhether or not the extracted 3D object candidate is a pedestrian.

Next, FIG. 3 illustrates the location information generation unit 500for a pedestrian. A location information calculation unit 510calculates, for a detected pedestrian, an average value of locationinformation utilizing an average value of parallax images of thepedestrian in a frame. An average value of parallax images excludingparallaxes largely different from the parallax at a short distance andparallax of a distant view may be calculated so that background parallaxinformation is not included.

Next, recalculation of location information considering accuracyinformation is performed in an accuracy-information-considered locationinformation recalculation unit 520. Although the detail will bedescribed in the description of a process for generating locationaccuracy information, a region of a noise factor for the pedestrian isdetermined by the location accuracy information generation unit 400. Inthe rectangular frame used for the pedestrian detection, a region whichseems to be a pedestrian and a region which includes other 3D objectsand is likely to be an error factor are distinguished. The informationof the region which seems to be a pedestrian in the detection frame isacquired from the location accuracy information generation unit 400, andthe accuracy-information-considered location information recalculationunit 520 searches parallax information by concentrating on the regionwhich seems to be a pedestrian to again acquire the locationinformation. Thus, a pedestrian location having higher accuracy isextracted.

Next, the acquired pedestrian location is analyzed in a time-seriesmanner, and the destination of the pedestrian is predicted by a movementinformation prediction unit 530. This will be described with referenceto an upper chart in FIG. 17. If the pedestrian location can be acquiredtwo frames before the current frame, it is supposed that the pedestrianlocation from T-2[frame] to the current T[frame] can be acquired. Inthis case, the direction of movement of the pedestrian can be predictedby utilizing the pedestrian location information from T-2 to T[frame].An upper right broken arrow in FIG. 17 indicates the predicted movementof the pedestrian predicted from the pedestrian location.

However, in actuality, if the movement direction is predicted in themanner described above, i.e., if the movement prediction is performed byusing information, as it is, in which an error is caused on thepedestrian location because a part of the vehicle is in the pedestriandetection frame in T-2[frame], there may be the case in which wrongprediction is performed such that the pedestrian moves in the upperright direction indicated by the upper right broken arrow in FIG. 17,although the pedestrian actually moves just sideway as indicated by asolid right arrow in FIG. 17.

In view of this, the movement prediction is performed by using aninstantaneous value of the location accuracy information acquired by thelocation accuracy information generation unit 400. As illustrated inmovement prediction (a) using the location accuracy information in thelower-right part in FIG. 17, the location with low accuracy as a resultof using the location accuracy information is not used as the data forperforming the movement prediction. As described above, it is determinedwhether or not a location is used as the data for the movementprediction according to location accuracy information. Thus, data havinglow location accuracy which is likely to have a large error isdetermined not to be used as the data for the movement prediction,whereby the location data having a large error is not used for themovement prediction as illustrated in (a), and therefore, the actualmovement and the predicted movement can be obtained with less error.

In addition, instead of the method for determining whether or not acertain location is used according to a level of location accuracy, sucha method may be used in which, when accuracy is low, prediction inconsideration of low accuracy is conducted.

As illustrated in lower-left FIG. 17(b), when accuracy is predicted tobe high due to high location accuracy information, a high value is votedto one point, and when accuracy is predicted to be low due to lowlocation accuracy information, low values are voted on the periphery,not on one point. According to this, a voting result in which a votingrange is increased according to the magnitude of an error is obtained asillustrated in FIG. 17(b). In the movement prediction, a line connectingthese points is searched, and a line having the highest votes thereon issearched. According to this process, the movement prediction in which anerror in a location is considered based on the accuracy information isperformed, whereby movement prediction with less error can beimplemented.

By utilizing the location accuracy information as described above, evenin the case in which the accuracy of the location itself cannot beimproved, the calculation for the movement prediction is performed bygiving priority to the information having high location accuracy intime-series information, whereby the accuracy in the movement predictioncan be enhanced.

According to the effect of the present embodiment, not only for thelocation accuracy of a pedestrian who is rushing out but also for thelocation accuracy of a pedestrian who is crossing or a pedestrian who iswalking on a walking path, the magnitude of an error applied to alocation by 3D objects or background around the pedestrian can berepresented by the location accuracy information, whereby movementprediction or the like can more accurately be calculated. In addition,if the location accuracy information is extremely low, there may be amethod in which the instantaneous location information thereof is notused, because this information cannot be used for control, although apedestrian can be detected by this information. Thus, the movementprediction from which an influence due to an error is reduced can beimplemented.

In addition, the movement information prediction unit 530 not onlypredicts movement information but also filters time-series instantaneousvalue location accuracy information and sets this information as currentlocation accuracy information. The current location accuracy informationindicates what location accuracy information is used in a time-seriesmanner to obtain the result of action prediction. As illustrated in thecalculation equation in FIG. 7, the current location accuracyinformation is calculated through filtering by using the locationaccuracy information of an instantaneous value. When the locationaccuracy information generation unit outputs the location as a result ofthe correction of the location accuracy itself, the location accuracyinformation for the corrected result is outputted as the locationaccuracy information of an instantaneous value.

Next, the outline and the components of the location accuracyinformation generation unit 400 will briefly be described with referenceto FIG. 4. As illustrated in FIG. 4, the location accuracy informationgeneration unit 400 acquires information pertaining to location accuracyfrom various information items, finally acquires information pertainingto the location accuracy of a pedestrian candidate location by anaccuracy information acquisition unit 490, and delivers the acquiredinformation to a pedestrian detection unit 300. In the configuration ofdelivering only location accuracy information, the similar effect can beobtained by directly delivering the information to thedisplay/alert/control unit.

Herein, it is to be noted that, since the location informationgeneration unit 500 is supposed to recalculate the location informationby an accuracy-information-considered location information recalculationunit 520 on the basis of the information acquired by the locationaccuracy information generation unit 400, the information pertaining tothe pedestrian location, including the accuracy information, isdelivered to the location information generation unit 500. The detail ofthe process in the location accuracy information generation unit 400will be described later.

Finally, the display/alert/control unit 600 will be described withreference to FIG. 5. As illustrated in FIG. 6, a TTC calculation unit610 calculates a traveling course of the vehicle by using the speed ofthe vehicle and the yaw rate. Then, based on the movement predictioninformation of the pedestrian which has been detected and travelingcourse prediction information of the vehicle, an intersection pointwhere two action prediction paths intersect each other is extracted, anda TTC taken for the vehicle to move to this point is calculated. In thiscase, time is calculated as TTC, regardless of whether or not thevehicle will collide with the pedestrian. As for a pedestrian who isstopped without moving, the location of the pedestrian is considered tobe a point, and only when the distance between a line of a travelingcourse of the vehicle and the location where the pedestrian is stoppedfalls within a certain distance, the TTC (time-to-collision) iscalculated. As described above, the TTC calculation unit 610 calculatesTTC in the case in which it is supposed that the vehicle collides withthe detected pedestrian. Notably, a pedestrian or the like with whichthe vehicle will not obviously collide based on the direction ofmovement is outside the target for control and alert, so that thecalculation therefor may not be performed.

Next, the place where the pedestrian is supposed to be present after thecalculated TTC seconds is calculated to determine collision likelihood.FIG. 6 shows one example of the calculation. This is such a case inwhich a pedestrian crosses the traveling course of the vehicle, thepedestrian and the vehicle both moving straight. Since the time TTCbefore the vehicle reaches the depth where the pedestrian is present hasbeen obtained, the pedestrian location after the TTC seconds isestimated. Thus, the collision likelihood is determined on the basis ofthe pedestrian location, in consideration of the location of the centerof the vehicle and an offset width α of the pedestrian after the TTCseconds. Such cases may be considered in which the pedestrian hasalready crossed the lane on which the vehicle travels or the vehiclegoes through before the pedestrian crosses because the movement speed ofthe pedestrian is low. Therefore, the location of the pedestrian andspeed information are significant for the collision likelihood.

The collision likelihood for a pedestrian who is not moving is similarlydetermined here. To perform determination of whether it is used for thevehicle control, the collision likelihood of the pedestrian presentwithin the offset width α from the predicted course of the vehicle isdetermined.

Then, as for the vehicle which is determined to have collisionlikelihood by the TTC calculation unit which determines the collisionlikelihood, location accuracy information after the TTC seconds iscalculated, and the output content according to the accuracy isdetermined by an output determination unit 620. When the accuracy ishigh, the control to avoid collision is more positively executed, andwhen the accuracy is low on the contrary, the control is changed to lowvehicle control or changed only to issue an alarm. This is because, ifvehicle control is performed by mistake, for example, sudden braking isapplied even though the vehicle is not on a collision course, thevehicle is at a risk of being hit by a vehicle traveling behind thevehicle.

The calculation process is performed in which the accuracy of thepedestrian location after the TTC seconds is reduced in proportion tothe length of the time-to-collision TTC from the current locationaccuracy information as illustrated in FIG. 8. This is because, thelonger the length of the TTC is, the more likely it is that thepredicted location is wrong, i.e., the more likely it is that thedirection of movement of the pedestrian is changed or that the movementspeed is changed. Therefore, the collision location accuracy informationafter the TTC seconds is predicted by reducing the accuracy of thecurrent pedestrian location in proportion to the length of the time.

However, the vehicle control is not performed under the condition inwhich, if a driver applies brakes just after the moment a pedestrian ispresent on the collision course, the collision can completely beavoided. The vehicle control is not performed for a target object forwhich the collision is clearly avoided by applying brakes by the driver.Otherwise, sudden braking is applied in a scene unexpected by thedriver, and in the worst case, it can be considered that the vehicle ishit by a following vehicle. Even if the vehicle is not hit by thefollowing vehicle, when controls considered to be unnecessary by thedriver are frequently performed, such controls are unnecessary for thedriver. Sudden braking is applied only when the collision is difficultto be avoided unless the brake control is performed at this timing. Inactuality, the time-to-collision or the location accuracy informationupon collision can also be calculated. However, in the presentcircumstances, supposing that the driver applies brakes in advance, itis determined that there is no collision likelihood, and the locationaccuracy information is determined to be zero which is the lowest.

Next, the output determination unit 620 determines the output content onthe basis of the collision location accuracy information calculated bythe TTC calculation unit 610 and the offset width α from the predictedcourse of the vehicle.

FIG. 9 illustrates control levels of the vehicle. The control level ischanged according to the offset width or the location accuracyindicating the location of the pedestrian at the time at which thecollision is predicted to occur. It is considered that, the higher theaccuracy is and the smaller the offset width is, the higher thecollision likelihood is, and therefore, the control level is increasedto a higher level. As illustrated in FIG. 9, in a control level 1,so-called control is not performed, and the presence of a pedestriannear the vehicle is only displayed on a display with no sound. Only thedisplay is provided so that the driving operation by the user isinterrupted if the detected location or predicted location of apedestrian is wrong.

Next, in a control level 2, considering that there is collisionlikelihood, but such possibilities are presumed in which the predictedlocation of the pedestrian is deviated, the vehicle passes near thepedestrian, etc., the acceleration of the vehicle is suppressed, andemergency braking is prepared. However, the control content forinterrupting the normal driving operation by the driver is not provided.In a control level 3, emergency braking is not applied, but thecollision likelihood is determined to be extremely high, so that thepresence of the pedestrian is notified to a user with an alarm, andpreliminary preparation for collision avoidance is executed. The brakehydraulic pressure is increased to increase the response speed when theuser operates the brake, and the hydraulic pressure is increased andacceleration is suppressed, to more quickly activate the emergency brakewhen the location accuracy of the pedestrian is improved. Finally, in acontrol level 4, it is considered that the vehicle will definitelycollide with the pedestrian, and therefore, braking is applied to bringthe vehicle to an emergency stop, and notifies the driver of thepresence of the pedestrian by issuing a sound.

The output determination unit 620 determines the control level on thebasis of the predicted location accuracy and the offset width of thepedestrian illustrated in FIG. 10. As the offset width, which is thedifference between the predicted course of the vehicle and the predictedlocation of the pedestrian, is increased, the collision likelihoodbetween the pedestrian and the vehicle is reduced, so that the controllevel is lowered. In addition, the same is applied to the predictedlocation accuracy. Even if the collision with the vehicle is definitebased on the predicted location of the pedestrian, it is considered thatan error is caused on the pedestrian location itself, and an outputcontent in the lowered control level is determined.

An output unit 630 indicates an operation content respectively to adisplay unit 631, an alert unit 632, and a control unit 633, byutilizing the determination result. The display unit 631 provides adisplay, or displays the presence of a pedestrian on a vehicle meterpanel section. The location relation between the pedestrian and thevehicle may be displayed on the display in a simplified manner. Thealert unit 632 issues a sound notifying that the collision likelihoodwith the pedestrian is high on the basis of the indication from theoutput unit. The control unit 633 transmits a command to stop thevehicle to actually apply the brake.

Next, the detail of the location accuracy information generation unit400 will be described with reference to FIG. 4.

<Density Calculation Unit 410>

As illustrated in FIG. 11, the density calculation unit 410 calculatesthe density of effective parallaxes in the frame of the image of thepedestrian candidate extracted by the pedestrian detection unit. Herein,the effective parallax means a parallax pertaining to the part of theobject to which stereo matching has been performed. Further, the stereomatching indicates that the same objects to be imaged or correspondingparts or common parts of the object, included in a plurality of parallaximages, are associated with one another through comparison of parallaximages. According to this process, effective parallaxes which areparallaxes pertaining to the objects to be imaged or corresponding partsor common parts of the object, included in the parallax images, areobtained.

Matching of images in small areas including pedestrian candidates inleft and right stereo images is performed. When the same object havingan image characteristic amount is present in the areas, the matching issuccessful because the same object is viewed on the locations deviatedfrom each other by the parallax in the left and right images andtherefore, the image characteristics are similar. In this case, thevalue in the small area in which matching is successful is specified asan effective parallax, and the value in the small area for which amatching place is not found even by searching on the images captured bythe left and right cameras and therefore, matching is in failure, isspecified as an ineffective parallax.

The following equation in which the number of effective parallaxes inthe frame are divided by the area of the frame is specified as aneffective parallax density. It is considered that, the higher thedensity is, the larger the number of effective parallaxes in the frameis, so that the pedestrian location is reliable.Effective parallax density=number of effective parallaxes in frame/framearea

In addition, it may be configured such that, from among the effectiveparallaxes in the frame in the above equation, only the parallax valuesexisting near the pedestrian location extracted in a simplified mannerare concentrated, and the effective parallax density is calculated onthe basis of the values excluding the parallax of the background regionwhich is in the frame and has an effective parallax. Meanwhile, in thiscase, it is not so difficult to count the parallax of the backgroundwith a clearly different parallax value as being discounted. However, itis difficult to exclude a 3D object which is in the frame and at a shortdistance from the pedestrian, because of close parallax values.

<Peripheral Separation Degree Calculation Unit 420>

FIG. 12 illustrates the outline of the calculation of a peripheralseparation degree. In extracting the location accuracy of a pedestrian,if other object is included in the frame from which the pedestrian isextracted, it is highly likely that an error is caused on the pedestrianlocation due to an influence of the parallax value of the other 3Dobject. In view of this, it is checked whether or not there is another3D object present around the pedestrian. With this, it is checkedwhether or not parallax values other than that of the pedestrian becomean error factor in extracting the pedestrian location.

As illustrated in FIG. 12, the detected pedestrian frame is enlargedhorizontally, and all effective parallaxes in the region are projectedin the vertical direction of the image. In this case, in addition to thenarrowing to effective parallaxes, a pixel having a parallax valuelargely distant from the pedestrian location which has been roughlycalculated may be specified as background, and may be discounted. Asdescribed above, the separation degree from peripheral objects iscalculated by using an effective parallax histogram projected in thevertical direction of the image.

A histogram total M within a range of ±xdis on the peak of the histogramin the pedestrian frame is calculated. Then, positions of valleys whichare low and located on the left and right of the peak of the histogramare searched. A histogram total V1 within a range of ±xdis on the leftvalley and a histogram total V2 within a range of ±xdis on the rightvalley are calculated.

M on the peak and V1 and V2 on the left and right valleys in FIG. 12 arecompared. As the valleys are lower with respect to the peak, it isconsidered that the separation degree from the periphery is high. As theseparation degree is high, it is considered that the possibility of theinfluence of parallax information of other object, which is an errorfactor, on the pedestrian location is low.

The specific equation for calculating the separation degree is asfollows.Separation degree=(V1+V2)/(2×M)

FIGS. 13(A) to 13(C) illustrate an example in which the separationdegree is high and an example in which the separation degree is low, inactuality. FIGS. 13(A) to 13(C) illustrate one example indicating that,when the method is used in which an average of parallaxes in the frameis determined to be a pedestrian location, an error is caused because anobject having a different parallax is included in the detection frame ofthe pedestrian.

In a scene (A) on the left in FIG. 13, about a half of the body of apedestrian who is about to rush out from a vehicle can be seen, so thatpedestrian recognition could be enabled on the basis of the shape as thepedestrian. Therefore, when pedestrian detection is immediatelyperformed with only the parallax in the frame being used, the parallaxfor the vehicle side, or the parallax for the vehicle side including apart of the vehicle rear when seen from different angles, is regarded asthe parallax for the pedestrian by mistake, and with this state, thepedestrian location is calculated as illustrated in the left view on themiddle in FIG. 13. Therefore, a large error is caused on the parallaxand depth obtained by averaging. When the separation degree iscalculated under this condition, a histogram for parallaxes indicatingthat little gap is present between the pedestrian and the vehicle isobtained as illustrated in the lower-left part in FIG. 13, and so, theseparation degree is low. When the separation degree is low, theaccuracy information acquisition unit 490 considers that there is a highpossibility of the location accuracy being low, in consideration of thecondition that false parallax is prone to be included.

A scene (B) is intermediate between a scene (C) and the scene (A),wherein the separation degree is also intermediate. The scene (C) willbe described. In the scene (C), the pedestrian is away from the vehiclebody, so that the position of the peak and the positions of the valleysare clearly recognized in the histogram. Therefore, it is clearlyunderstood that there is no 3D object around the pedestrian, and it isdetermined that the separation degree in this case is high and thepedestrian location accuracy is also high.

Further, a method for improving the location accuracy of the pedestrianusing the separation degree will also be described. Firstly, in thestate in which the separation from the periphery is clear as illustratedin FIG. 13(C), it is difficult to consider that the error factor of theparallax information of an object other than the pedestrian is caused.Therefore, it is construed that the pedestrian location is calculatedwith high accuracy and the location correction is unnecessary.

However, in the case of the scene (B), it is found that there is aportion where the vehicle and the pedestrian overlap each other in theframe in which the pedestrian is detected. Therefore, the pedestrian andthe vehicle are separated from each other using the valley position tocalculate the accuracy of the pedestrian location.

Actually, it is considered that, when V1/M or V2/M is not less than acertain threshold, a detection frame including both the periphery and anobject is likely to be formed. FIG. 14 illustrates the outline inactually performing correction. It is considered that the right sidefrom the valley position on the center position of V1 does not need thecorrection for separation, because V1/M is not less than the threshold,V2 is originally outside the frame, and V2/M is obviously not more thanthe threshold. Therefore, distance information is generated by using aparallax image in a region between the left end of the frame being thevalley position of V1 and the right end which is the original right endof the pedestrian detection frame, in order to improve the accuracy ofthe location information.

<Original Texture Analysis Unit 430>

Texture analysis of an original image is performed, and segmentation isconducted on the current image. An enlarged region including theperiphery of the pedestrian detection frame of the original image issegmented into four, and uniformity of images in four segmented regionsis evaluated. If the images are determined to be uniform, they areconsidered to be a portion of the same segmentation. If the images aredetermined to be non-uniform, segmentation in four regions is repeated.Thus, an input image is segmented into a tree structure, and similarityof adjacent segmented images corresponding to a termination node of thetree structure is evaluated. If the images are determined to be similar,the adjacent regions are considered to belong to the same segmentation,and are combined together. This process is repeated until there are noregions to be combined.

As described above, segmentation is conducted on an original imagebasis, and an average of parallaxes for each segmentation is obtained.The region where a parallax greatly varies is originally discounted. Aregion containing objects which are relatively close to each other andhave parallax values close to each other, such as a parked vehicle and apedestrian, and segmented into different segmentation is referred to asa target. In this case, it is possible to improve the location accuracyof the pedestrian by calculating distance information by excluding theparallax information of the segmentation region which is considered tobe other than the pedestrian in the pedestrian detection frame.Particularly, in this segmentation, for the same object such as anartificial material, segmentation is well performed for an object whichis more likely to have the same texture. Therefore, the segmentation isutilized to exclude the parallax information in segmentation of anartificial material, such as a building or a vehicle, included in thepedestrian detection frame, rather than whether or not the segmentationfor the pedestrian is in success.

The original image texture analysis unit 430 may be used only when it isfound that the separation from the periphery is not achieved by theperipheral separation degree calculation unit 420. In addition, thissegmentation provides the calculation indicating that the accuracy ofthe location information is low, as there are a lot of anothersegmentation included in the pedestrian detection frame.

Further, when it is found that the separation from the periphery is notachieved by the peripheral separation degree calculation unit 420, andan original image seems to be not segmented well into segmentation dueto less texture, the location accuracy may be determined to be lowbecause of low parallax density in the pedestrian frame, consideringthat the edge intensity inside is extremely low.

<Rushing Out Region Priority Unit 440>

This unit may be used only when it is found that the separation from theperiphery is not achieved by the peripheral separation degreecalculation unit 420. In the case in which an object is obviously theone separated from behind a 3D object, it is likely that a pedestriandetection frame is formed in which a part in the rushing out directionis a portion of the body of the pedestrian and a portion of the 3Dobject which is an object shielding the body of the pedestrian on theside opposite to the rushing out direction is included. In such case,the pedestrian condition is classified into three categories: apedestrian moving to the right, a pedestrian moving to the left, and apedestrian who is not moving, wherein the categorization may beperformed with an error being caused. If it is found to be a pedestrianwho is now moving and rushing out from behind a 3D object, a method forimproving accuracy is considered by using only the parallax of the halfside of the pedestrian in the direction of movement. This process isperformed only when the separation between the pedestrian and the 3Dobject blocking the pedestrian is not in success even by the separationdegree or the texture analysis. Although the 3D object which is ablocking object and the pedestrian are not actually separated from eachother, the possibility of the decrease in the proportion of the blockingregion is increased, and the possibility of improvement in the locationaccuracy is high. Thus, this process is executed.

<Accretion Analysis Unit 450>

It can be determined by the original image texture analysis that thelocation accuracy is reduced when there is a little parallax informationin the detection frame. However, the case in which the location accuracyis reduced despite parallax information being present may actuallyoccur.

Next, the accretion analysis unit 450 will be described with referenceto FIGS. 15(a) to 15(c). The outline of the accretion analysis unit thatdetects accretion, such as mud, which blocks the background will bedescribed with reference to FIGS. 15(a) to 15(c). It is a logic fordetecting a region where there is mud which makes it difficult torecognize the background, the luminance is continuously lowered ascompared to the periphery, and the luminance variation is small. Theaccretion analysis unit 450 divides the image region of the capturedimage into a plurality of blocks A (x, y) as illustrated in FIG. 9(b).

Next, the accretion analysis unit 450 detects the luminance of eachpixel in the captured image. Then, the accretion analysis unit 450calculates the luminance total I_(t)(x, y) of respective pixels includedin a block A(x, y) for each block A(x, y). The accretion analysis unit450 calculates, for each block A(x, y), the difference ΔI(x, y) betweenI_(t)(x, y) calculated for the captured image in the current frame andI_(t-1)(x, y) similarly calculated for the captured image in theprevious frame.

The accretion analysis unit 450 detects a block A(x, y) in which ΔI(x,y) is smaller than the peripheral blocks, and increments the score SA(x,y) corresponding to the block A(x, y) by a predetermined value, e.g., by1.

After performing the above determination for all pixels in the capturedimage, the accretion analysis unit 450 acquires an elapsed time to fromthe initialization of the score SA(x, y) of each block A(x, y). Then, anaccretion detection unit 240 calculates a time average SA(x, y)/tA ofthe score SA(x, y) by dividing the score SA(x, y) of each block A(x, y)by the elapsed time tA.

In the state in which accretion is deposited as described above, a falseparallax is generated on the accretion and the background, or parallaxinformation itself cannot be obtained, in some cases. Therefore, if theaccretion region in the pedestrian detection frame is not less than acertain threshold, location accuracy information is lowered according tothe proportion thereof.

<Light Source Analysis Unit 460>

As illustrated in FIG. 16, there arises a problem such that, due tobacklight or reflection, luminance becomes high on a screen, by which aregion where background is invisible is generated. If light reflectionor the like occurs, a false parallax may be generated. If backlightoccurs, a parallax cannot be obtained from the periphery. Therefore, ina high-luminance region illustrated in FIG. 16, a possibility in which aright parallax is obtained is determined to be low, and thus, locationaccuracy information is lowered according to the proportion of thehigh-luminance region in the pedestrian detection frame.

<Accuracy Information Acquisition Unit 490>

Location accuracy information of an instantaneous value is acquired byusing the density calculation result, the separation degree calculationresult, the texture analysis result, whether the rushing out regionpriority is used or not, the accretion analysis result, and the lightsource analysis result. When the location correction is performed in theseparation degree calculation, the accuracy information acquisition unit490 acquires location accuracy information to which correction has beenperformed, not the initial location accuracy information.

Next, a process flow for preventing collision with a pedestrian will bedescribed with reference to FIG. 18.

In step S01, the stereo camera imaging unit 100 captures an environmentahead of the vehicle.

In step S02, the parallax image generation unit 200 executes a stereomatching process using the image captured by the stereo camera imagingunit 100 to generate a parallax image.

In step S03, the 3D object extraction unit 310 extracts 3D objectcandidates from the parallax image of the current frame.

In step S04, the tracking unit 320 tracks the 3D object candidatesindependently extracted for each frame in step S03, by using thelocations of the 3D object candidates in the current frame and at leasttwo or more of the locations of the 3D object candidates in the previousframe, the speed information of the 3D objects, the vehicle behavior,and the like. By tracking the 3D object candidate in a time-seriesmanner, a noise factor, such as an emergence of a 3D object candidate inonly one frame, can be eliminated.

In step S05, 3D object candidates which are falsely extracted areeliminated by tracking, and a pedestrian candidate 3D object with a sizewhich can be a pedestrian candidate is selected from them by therecognition unit 330 in a simplified manner. It is determined that anobject which is too large or too small cannot be a pedestrian, and theprocess proceeds to step S07. For a 3D object which can be a pedestriancandidate, the process proceeds to step S06.

In step S06, pedestrian recognition is executed by the recognition unit330 only for the pedestrian candidate 3D object which can be tracked.The pedestrian recognition is executed by using an edge image generatedfrom the current image, parallax information, etc. The pedestriancandidate 3D object which has been recognized as being not a pedestrianis additionally registered as a 3D object candidate which is determinedto be other than the pedestrian in step S07. For the pedestriancandidate 3D object which is recognized as a pedestrian by therecognizer, the process proceeds to step S08. Note that, in the presentembodiment, detection of a vehicle or a sign other than a pedestriancandidate is not handled. Therefore, they are extracted as 3D objectcandidates other than a pedestrian, and the process afterward will beomitted.

In step S08, for the object which has been recognized as a pedestrian,the parallax value in the frame upon extraction of the 3D objectcandidate is observed to generate the pedestrian location by thelocation information calculation unit 510.

In step S09, after the generation of the location information, thelocation accuracy information generation unit 400 acquires the accuracyinformation of the pedestrian location or determines whether or not itis possible to improve the accuracy of the pedestrian location. If thelocation accuracy information generation unit 400 determines that it ispossible to improve the accuracy, the process returns to step S08 wherethe location information is again generated by theaccuracy-information-considered location information recalculation unit520. The process proceeds to step S10 when the location accuracyinformation generation unit 400 determines that it is impossible toimprove the accuracy, that is, determines that there is no need toimprove the accuracy because the accuracy is relatively good, and whenit is determined that the accuracy is not good but there is no method toimprove the accuracy.

In step S10, the movement information prediction unit 530 predicts thedestination of the pedestrian using the location information and thelocation accuracy information of the pedestrian.

In step S11, the collision between the pedestrian and the vehicle basedon the prediction of the movement of the pedestrian and the predictionof the vehicle behavior. When they are likely to collide against eachother, display, alert, and control are executed according to thelocation accuracy information and the pedestrian location.

As described above, the object detection device according to the presentembodiment is conducive to carrying out control in an appropriate manneraccording to the surrounding environment, with consideration to theaccuracy of locations of objects.

In addition, the object detection device extracts a 3D object from adistance image generated by using a stereo camera mounted on a vehicle,acquires the location information of the extracted 3D object, andanalyzes in detail the accuracy information of the acquired locationinformation and whether or not the location information can becorrected. When the location information can be corrected, the objectdetection device again acquires the location information and thelocation accuracy information, estimates the location and speed of the3D object by using these results, and more appropriately determines thecollision likelihood from the behavior of the vehicle, thereby beingcapable of reducing a delay in the control or the occurrence of falsecontrol.

Further, it is not easy to always accurately specify the location of anobject. However, using only the accurate location as a detection resultfor performing the control according to the surrounding environment isnot suitable for the recognition of the surrounding environment. In viewof this, the object detection device according to the present embodimentuses location information in consideration of the detection state of thelocation of the detected object. The vehicle can appropriately becontrolled according to the accuracy of the location information.

Note that the object detection device according to the present inventionis not limited to the embodiment described above, and variousmodifications are possible without departing from the spirit thereof.

For example, while the above embodiment describes that an object to bedetected is mainly a pedestrian, it is not limited thereto. An object tobe detected may be a bicycle, a motorbike, a four-wheel vehicle, ananimal, or other moving bodies. Besides, an object to be detected may bea columnar body such as an electric pole or a signal, or a fixed body ona ground such as a wall.

REFERENCE SIGNS LIST

100 . . . stereo camera imaging unit, 200 . . . parallax imagegeneration unit, 300 . . . pedestrian detection unit, 310 . . . 3Dobject extraction unit, 320 . . . tracking unit, 330 . . . recognitionunit, 400 . . . location accuracy information generation unit, 410 . . .density calculation unit, 420 . . . peripheral separation degreecalculation unit, 430 . . . original texture analysis unit, 440 . . .rushing out region priority unit, 450 . . . accretion analysis unit, 460. . . light source analysis unit, 490 . . . accuracy informationacquisition unit, 500 . . . location information generation unit, 510 .. . location information calculation unit, 520 . . .accuracy-information-considered location information recalculation unit,530 . . . movement information prediction unit, 600 . . .display/alert/control unit, 610 . . . TTC calculation unit, 620 . . .output determination unit, 630 . . . output unit, 631 . . . displayunit, 632 . . . alert unit, 633 . . . control unit

The invention claimed is:
 1. An object detection device comprising: anobject detector configured to generate parallax information from aplurality of parallax images acquired from a plurality of cameras;detect objects contained in the parallax images; generate locationinformation about the objects, on the basis of the parallax information;and generate location accuracy information pertaining to the accuracy ofthe location information, on the basis of a characteristic of theparallax information, generate the location information on the basis ofan effective parallax, the effective parallax being a parallaxpertaining to a part of the objects for which stereo matching has beensuccessfully performed in an object region corresponding to the objects,and generate the location accuracy information on the basis of ahistogram obtained by projecting an amount of effective parallaxescontained in an enlarged region including the object regioncorresponding to the objects and an adjacent region adjacent to theobject region on one axis along an area between the object region andthe adjacent region.
 2. The object detection device according to claim1, wherein the location accuracy information is generated on the basisof a proportion between an area of the object region containing theobjects and an area of an effective parallax region which is included inthe object region and from which effective parallaxes are obtained. 3.The object detection device according to claim 1, wherein the locationaccuracy information is generated on the basis of characteristic pointinformation of a corresponding region corresponding to the object regioncontaining the objects in at least a single parallax image in theplurality of parallax images.
 4. The object detection device accordingto claim 1, wherein, the effective parallaxes are in the object regioncontaining the objects and correspond to the distance of the objects,and the location information and the location accuracy information aregenerated on the basis of the effective parallaxes.
 5. The objectdetection device according to claim 1, wherein when the accuracy of thelocation information of the objects is low, and the direction ofmovement of the objects determined from a time-series location of theobjects makes an approach to a course of the object detection device,the location information about the objects and the location accuracyinformation are generated on the basis of parallax information for aforward section in the direction of movement of the objects, out ofeffective parallaxes being parallaxes pertaining to a part of theobjects to which stereo matching has been performed in an object regioncorresponding to the objects.
 6. The object detection device accordingto claim 1, wherein the object detector uses the cameras to detect apedestrian, and when the pedestrian is imaged as overlapping otherobject in an optical axis direction of the cameras, the pedestrian andthe other object are separated from each other, and the locationinformation and the location accuracy information of the pedestrian aregenerated on the basis of the parallax information on a portion, of thepedestrian, separated from the other object.
 7. The object detectiondevice according to claim 1, wherein the object detector is configuredto generate the location accuracy information on the basis of a degreeof overlap between an object region containing the objects and ahalation region where halation occurs due to light.
 8. The objectdetection device according to claim 1, wherein the object detector isfurther configured to generate the location accuracy information on thebasis of a degree of overlap between an object region containing theobjects and an accretion region where accretion is deposited to blockimaging regions of the cameras.
 9. The object detection device accordingto claim 1, wherein the object detection device is mounted on a vehicle,and is configured to change control level of the vehicle on the basis ofthe location information and the location accuracy information.
 10. Anobject detection device comprising: an object detector configured to:generate parallax information from a plurality of parallax imagesacquired from a plurality of cameras; detect objects contained in theparallax images; generate location information about the objects, on thebasis of the parallax information; and generate location accuracyinformation pertaining to the accuracy of the location information, onthe basis of a characteristic of the parallax information, wherein thelocation information is generated on the basis of an effective parallax,the effective parallax being a parallax pertaining to a part of theobjects to which stereo matching has been successfully performed in anobject region corresponding to the objects, and the location accuracyinformation is generated on the basis of a proportion between an area ofthe object region containing the objects and an area of an effectiveparallax region which is included in the object region and from whicheffective parallaxes are obtained.
 11. An object detection devicecomprising: an object detector configured to: generate parallaxinformation from a plurality of parallax images acquired from aplurality of cameras; detect objects contained in the parallax images;generate location information about the objects, on the basis of theparallax information; and generate location accuracy informationpertaining to the accuracy of the location information, on the basis ofa characteristic of the parallax information, wherein when the accuracyof the location information of the objects is low, and the direction ofmovement of the objects determined from a time-series location of theobjects makes an approach to a course of the object detection device,the location information about the objects and the location accuracyinformation are generated on the basis of parallax information for aforward section in the direction of movement of the objects, out ofeffective parallaxes being parallaxes pertaining to a part of theobjects to which stereo matching has been performed in an object regioncorresponding to the objects.
 12. An object detection device comprising:an object detector configured to: generate parallax information from aplurality of parallax images acquired from a plurality of cameras;detect objects contained in the parallax images; generate locationinformation about the objects, on the basis of the parallax information;and generate location accuracy information pertaining to the accuracy ofthe location information, on the basis of a characteristic of theparallax information, wherein the location accuracy information isgenerated on the basis of a degree of overlap between an object regioncontaining the objects and a halation region where halation occurs dueto light.
 13. An object detection device comprising: an object detectorconfigured to: generate parallax information from a plurality ofparallax images acquired from a plurality of cameras; detect objectscontained in the parallax images; generate location information aboutthe objects, on the basis of the parallax information; and generatelocation accuracy information pertaining to the accuracy of the locationinformation, on the basis of a characteristic of the parallaxinformation, wherein the location accuracy information is generated onthe basis of a degree of overlap between an object region containing theobjects and an accretion region where accretion is deposited to blockimaging regions of the imaging units.